A General Compression Scheme for Databases

Compression of databases not only achieves a reduction in storage space but can reduce overall retrieval times. Current schemes such as gzip and compress are impractical for the purposes of databases as they do not allow individual records to be retrieved. A recent compression scheme, sequitur, allows quick decompression of any individual section of the database, however it uses extravagant amounts of main memory during compression. We propose an extension to the sequitur algorithm that limits main memory usage during compression. We show that our new scheme achieves compression performance only slightly worse than sequitur while allowing massive reductions in memory usage.